Request changes. Logic is faithful to upstream + the internal GLM-OpenEvals impl (verified: both jsonl checksums, h5 byte-identity via sha256, prompt templates, extract_python_script, and previous_llm_code handling all match). Three things to fix:

🔴 Blocking

• Missing gold-step files. special_step_code() (sieval/community/scicode/prompts.py:102) reads data/{13.6,62.1,76.3}.txt, but .gitignore:156 (data/) excludes them — they're not committed and not on disk. Reproduced: all three raise FileNotFoundError, so problems 13/62/76 crash in infer() and pollute main-problem accuracy. reference_impl.notes claims these are vendored, but they aren't. Suggest inlining them as a .py module (like community/livecodebench/prompts/*.py) to avoid both the gitignore and wheel-packaging ([tool.pdm.build] declares no includes for data files).

🟠 Repro fidelity (blocks matching official numbers)

• Timeout. timeout=300.0 (sieval/tasks/scicode_0shot_gen.py:116) vs the official hardcoded 1800 in upstream test_generated_code.py. 6× shorter → spurious timeouts on slow numerical steps. Bump to 1800 or document why.
• Temperature. Official generation is greedy (temperature=0). Correctly not hardcoded here, but please note in the docstring/reference_impl.notes that reproducing official numbers requires temperature=0 in the model config. (with_background=False default is already correct — matches the official headline mode.)

Nits

• Remove the unused get_function_from_code / extract_function_name exports (sieval/community/scicode/__init__.py) — dead code; they read as "aligning with upstream" but nothing calls them (upstream only uses them on the resume-from-disk path, which this task doesn't take).
• Flesh out reference_impl.notes with the real deviations: h5 targets inlined + remote eval service (vs in-subprocess), special-step verbatim embed, and pipeline-fails-counted-as-unsolved accounting.
• Document the sandbox dependency sympy (injected by _cmp_upstream.py; not in any sieval dep group — must exist in the code-eval service image).
• _cmp_source() (sieval/community/scicode/harness.py:26) re-reads the file per tested step — wrap in functools.lru_cache.

Blocking = the gold files + timeout; the rest can be quick follow-ups.

Request changes. Logic is faithful and the prior round is fully addressed, but verifying against the runs in outputs/ surfaced one blocking issue plus a few nits.

🔴 Blocking — committed code diverges from upstream, and the reported number isn't from it

Special steps are embedded verbatim (sieval/tasks/scicode_0shot_gen.py:170-189, sieval/community/scicode/prompts.py:96-102), but upstream (gencode_json.py) runs get_function_from_code(gold, extract_function_name(header)) — which resolves to __init__ and extracts a bare method, dropping the class wrapper for 13.6/62.1 (and EnlargedBlock for 62). This isn't cosmetic:

• In the scicode-kimi run, problem 13's steps 13.8–13.15 all fail with NameError: Maxwell (extraction path). The committed verbatim code makes 13.8/13.14 pass (confirmed in scicode-kimi-p13; the model never re-defines Maxwell). On a 10-problem slice this moves main accuracy 20%→30%.
• The PR's without-background 37.8 comes from scicode-kimi, which used the old extraction path at T=0.0 — not the committed verbatim code, and not the "T=1.0 official hyperparams" stated for the with-bg number. There is no full-set run of the committed code in headline (without-background) mode.
• reference_impl.notes says reproduction needs temperature=0, but the validated with-bg number (49.3) used T=1.0; reference_impl.url points at upstream (extraction) while the code does verbatim. Task is status="stable".

Please: (1) re-run the full 65-problem set with the committed code in without-background mode and update the number, with a single stated temperature; (2) decide which reference the numbers target (public upstream = extraction, or the internal verbatim variant) and make code/url/notes consistent, flagging the special-step handling as a score-relevant deviation; (3) set status="experimental" until a full run of the committed code validates it.

🟠 Should fix

• postprocess (sieval/tasks/scicode_0shot_gen.py:243) opens the ~1 GB h5 synchronously per tested step inside async def, stalling the loop for concurrent samples. Read once per problem or await asyncio.to_thread(...).

⚪ Nits

• The evaluator caps memory at 1024 MB (Sample.memory_limit default, not overridden here); upstream's subprocess has no cap — memory-heavy numerical steps could OOM here but pass upstream. Worth a note in reference_impl.notes.
• .gitignore:192-197: the personal scicode-kimi*.yaml entries will ship to OSS as noise — prefer a glob or .git/info/exclude.
• MULTISTEP_TEMPLATE (sieval/community/scicode/prompts.py:29-52) has a trailing \n upstream lacks (only affects with_background=True).
• Dataset description says "80 problems / 338 sub-steps" (sieval/datasets/scicode.py:1,54); the headline test split is 65 / 288 (80/338 is test+dev).